[Report]AWS Well-Architected with Amazon OpenSearch Service#AWSSummit

This article is a session report of the AWS Summit Online session "AWS Well-Architected with Amazon OpenSearch Service" held on May 25, 2022.

Session overview

Title: AWS Well-Architected with Amazon OpenSearch Service

Summary: Amazon OpenSearch Service is a back-end data storage engine employed for a variety of use cases including log analytics, observability, and search. However, you need to ensure that the Amazon OpenSearch Service domain and the associated workload is AWS Well-Architected. In this session, we discuss best practices for using Amazon OpenSearch Service.

Speaker :  Muhammad Ali, Senior Analytics Specialist Solutions Architect AISPL.

 

Session Report

Amazon OpenSearch Service

Amazon OpenSearch Service is a managed service. OpenSearch makes it easy to deploy, operate, and scale OpenSearch legacy Elasticsearch clusters in the AWS Cloud. Amazon OpenSearch Service is the successor to Amazon ElasticSearch Service. With OpenSearch Service you can perform interactive log analytics, website search, real-time application monitoring, and more.

Working of OpenSearch Service

• Server, application, network, AWS, and other logs or application data is sent to OpenSearch Service in JSON documents.
• OpenSearch Service Indexes and converts data into instantly searchable format.
• Rest APIs, clients, and OpenSearch Dashboards can be used for Searching , Analyzing  and Visualizing Log Data.

 

Well-Architected OpenSearch Service

Sustainability

• OpenSearch Service partitions your data into shards.
• Configure OpenSearch with right shard count, Instance type, storage and Instance count.
• Right Sizing of OpenSearch is necessary.

Right Sizing

• Storage need : use the below formula to find the storage needs.
  • Storage needed = Source/day * 1.1 * 2 * retention * 1.15
• Shard count : calculate the shard count using the below formula.
  • Primary Shards = • Index size / target shard size
  • It is recommended to use 50 GB max for Logs and 20-30 GB for Search evaluation.
• Set a template : Set a template with a number of shards and replicas.

 

PUT _template/<template_name>
{
    "index_patterns": ["logs-*"],
    "settings": {
        "number_of_shards": 10,
        "number_of_replicas": 1
    },
    "mappings": {
        …
    }
}

 

• Adjust for usage :
  • vCPU = 1.5 * active shards
  • Use Primaries for queries.
  • Use Primaries and replicas for updates.
• Choose the node type : 
  • Choose the Instance type based on the requirements.
  • M6g - General Purpose and  Cost effective.
  • R6g - RAM speeds up searches and  Good start for log analytics.
• It is a Best Practice to Deploy, Monitor and Adjust configurations.

 

Cost Optimisation

• Use different Data nodes based on the use case.
• Dedicated Leader Nodes : used for cluster management.
• Hot Data Nodes : stores data and handles indexing and search requests.
• UltraWarm Data Nodes :
  • stores older, less frequently accessed data at 90% less cost.
  • These nodes are optimised caching and query nodes for indexes stored in S3.
  • Data in UltraWarm is immutable.
• Cold storage :
  • Cold storage is a fully decoupled/detached UltraWarm indices.
  • Provides a virtually unlimited online archive at near S3 prices.

 

Reliability

• Use Dedicated Leader Nodes.
  • Leader Nodes orchestrates and scales based on configuration.
  • Reference Link : Dedicated Leader nodes in Amazon OpenSearch Service
• Use Multi-zone Deployment.
  • Use 3 zones and 2 replicas.
  • Nodes are divided across zones and Data is replicated across zones.
• Use Cross-Cluster Replication.
  • With Cross-cluster replication, OpenSearch is highly available and It provides disaster recovery.
  • It supports Cross-account and cross-region.

 

Security

• You can Integrate OpenSearch with SAML and Cognito for OpenSearch Dashboards login
• Control access to the endpoint using IAM.
• Encrypt your data, in flight and at rest.
• To secure your data and dashboards, use OpenSearch fine-grained access control.
• Use a private endpoint to deploy into your VPC and security groups for traffic control.

 

Performance Efficiency

• Use the Auto-Tune feature of OpenSearch Service.
  • Auto-Tune uses performance and usage metrics from your OpenSearch cluster to suggest memory-related configuration changes.
  • It automatically maximizes efficiency and performance.
  • It improves cluster speed and stability.

 

Operational Excellence

• Setup the recommended CloudWatch alarms for Amazon OpenSearch Service.
• When a CloudWatch metric exceeds a specified value CloudWatch alarms perform an action.
• Reference Link : Recommended Alarms

 

Conclusion:

OpenSearch Service is a powerful analytical engine • For designing well architected OpenSearch applications, OpenSearch Service provides many features.

References : 

Amazon OpenSearch Service

Dedicated Leader nodes in Amazon OpenSearch Service

Recommended Alarms